Optimizing Document Similarity Detection in Persian Information Retrieval

Most data on the Web is in the form of text or image. Finding desired data on the Web in a timely and cost-effective way is a problem of wide interest. In the last several years, many search engines have been created to help Web users find desired information. In this paper we present a new technique to eliminate the affixes and their effects on recognizing similar Persian documents. Reviewing affixes’ rules and exceptions in Persian language, we extracted about 300 common inflectional suffixes and their combinations. We evaluate the effectiveness of eliminating the affixes from Persian texts on document similarity using four major document similarity approaches: Latent Semantic Indexing, Shingling, Vector Space Model, and Co-occurrence. Evaluation results demonstrate improvement in retrieval and detection of similar documents after eliminating affixes.